Functional Integration

 

Integration Patterns

Start | Previous | Next

Contents

Context

Problem

Forces

Solution

Resulting Context

Testing Considerations

Security Considerations

Acknowledgments

Context

Enterprise information systems contain a variety of applications that provide levels of interaction that range from simple to complex. The functions that are included in these systems offer a variety of capabilities and levels of access, from undocumented functions and silo applications to fully developed composite applications. In addition, the mechanism for invoking a function differs from one system to another.

Problem

How do you integrate information systems that were not designed to work together?

Forces

To correctly solve this problem, you need to consider the following forces:

• Most enterprises contain multiple systems that were never designed to work together. The business units that fund these information systems are primarily concerned with functional requirements rather than technical architectures. Because information systems vary greatly in terms of technical architecture, enterprises often have a mix of systems, and these systems have incompatible architectures.
• Many applications are organized into three logical layers: presentation, business logic, and data.
• Most commercial business applications developed in the last decade provide stable documented programming interfaces to allow access to the business functionality that is incorporated in the application. Software vendors provide these APIs specifically to support integration with external software.
• The functional interfaces provided by an application typically abstract from the underlying data representations so that they are more stable than the internal representations. For example, while a database schema might change in a new version of the software, the vendors keep the public APIs compatible with the previous version so that external applications accessing the API do not break.
• Accessing a function that resides in another application provides a natural extension of the application programming model that developers are used to. The semantics of accessing an external function that resides in another application are similar to making a local method call. This allows for natural integration with existing applications.
• Making direct updates to another application's data store through Data Integration bypasses all business and validation logic incorporated in the application's business logic layer. As a result, the risk of corrupting an application's data store is high.
• Exchanging messages between applications that include both data and behavior can provide for a more powerful conversation between systems. Instead of sending a message that only contains a customer's address, one system can instruct the other system to perform a specific function. For example, one system can instruct another system to validate the customer's address or to update a customer record with the customer's new address. More powerful conversations also allow an application to more specifically describe the functions and services it can provide to other applications.
• Many programming interfaces are programming language-specific and are not available remotely unless they have been developed on top of a specific remoting technology, such as Microsoft .NET Framework remoting or Common Object Request Broker Architecture (CORBA).

Solution

Integrate applications at the business logic layer by allowing the business function in one application (the source) to be accessed by other applications (the target), as shown in Figure 1.

Figure 1. Integrating applications at the business logic layer

For an external application to integrate with the source application through Functional Integration, the following two conditions must be met:

• The business function that you want must be available inside the source application's business logic.
• The API of the source application must be accessible remotely.

If the desired business function is not available, you have to modify the source application to make the function available. If modifying the source application is not feasible, you should add the new function outside the source application and then communicate with the application through another form of integration, such as Data Integration or Presentation Integration. This approach has fewer side effects and is generally available for most types of applications.

Many applications only expose their business functions as a local API that is dependent on a specific programming language such as C++ or C#. In those cases, a local adapter or middleware interface has to be created that translates incoming messages from other applications into local API calls. Likewise, results from API calls are translated back into messages. In most cases, such an adapter can be generic enough to support a variety of different functions without having to be modified for each individual function that you want to make available externally.

Functional Integration is very versatile because many different operations can be performed though an API. A functional interface can retrieve data, update data entities (change an address), or perform business functions (validate a credit card). In fact, one common use of Functional Integration is to ensure data consistency between applications [Ruh01].

Functional Integration is based on the interaction between the components that are described in Table 1.

Table 1: Functional Integration Components

Components	Responsibilities	Collaborators
Business logic	Executes local business functions	Middleware interface
Middleware interface	- Converts incoming messages into method invocations of functions that reside in the business logic - Converts return data back into messages that can be transported across the network		Business logic and remote application
Remote application	Consumes functions that reside in the application	Middleware interface

 

Resulting Context

After you decide to use Functional Integration, you must choose a particular kind of integration that is appropriate for your situation. Your choices are summarized by the following patterns:

• Distributed Object Integration
• Message-Oriented Middleware Integration
• Service-Oriented Integration (through XML-based Web services)

Distributed Object Integration

Distributed Object Integration is also known as instance-based collaboration because it extends the model of object-oriented computing to distributed solutions. Objects inside one application interact with objects in another remote application in the same way that they would interact locally with another object. This implies that the interaction occurs with a specific object instance and that the client application often manages the lifetime of the object it is accessing. This type of interaction usually seems natural to application developers, but it can result in a complex and tightly-coupled interaction model between the components. This tight coupling is not a problem as long as the components are part of a single distributed application. However, it is generally not a good choice when integrating multiple stand-alone applications.

Great examples of distributed component middleware are technologies such as .NET remoting, COM+, or CORBA. For more information, see the Remote Procedure Invocation pattern [Hohpe04] or the "Distributed Systems" chapter in Enterprise Solution Patterns Using Microsoft .NET [Trowbridge03].

Message-Oriented Middleware Integration

Message-Oriented Middleware Integration connects systems by using asynchronous message queues that are based on proprietary message-oriented middleware. The connected systems then communicate by using messages that contain small packets of data. Because the communication is asynchronous and durable, there is little chance of the messages being lost during network or system failure.

To share request/response type functionality, the consuming system must create a request message and send it by way of the message queue to the system that provides the functionality. The provider then takes the message from the queue, interprets it as a request, and processes it. Upon completion, the provider creates a response message and sends it back to the functional consumer by way of the message queue. Of course, not all functionality is shared by using a request/response style collaboration, but similar principles apply. For more information, see the Messaging pattern [Hohpe04].

Service-Oriented Integration

Service-Oriented Integration connects systems by enabling them to consume and provide XML-based Web services. The interfaces to these systems are described through Web Services Definition Language (WDSL) contracts. Systems interact with each other by using SOAP messages. SOAP messages are usually conveyed through HTTP by using XML serialization.

A Service Interface [Trowbridge03] exposes functionality as a Web service, and, a Service Gateway encapsulates the logic necessary to consume services (see Figure 2).

Figure 2. Using a service gateway and service interface to connect a Web service consumer and provider

Using Service-Oriented Integration increases interoperability by using XML and XML Schema as the basis of message exchange and by using SOAP as an extensible messaging framework. XML Schema provides for a type system that is portable between disparate technical architectures. In contrast, SOAP can be bound to a number of different transport mechanisms. For more information, see Service-Oriented Integration.

Choosing Between Alternatives

There are many factors to consider when choosing the kind of Functional Integration that is best for your particular requirements. Some of these factors include:

• Reliability and latency of the network between endpoints
• Interfaces exposed by your current systems
• Need for interoperability between disparate technical architectures
• Performance
• Fragility, if incompatible updates are introduced to any participant
• Expertise of the technical team
• Existing infrastructure

Choosing Distributed Objects

If your team is proficient with object-oriented development and if you use a platform infrastructure that offers a Broker such as .NET Framework remoting, Distributed-Object Integration can be fairly simple to implement. Although the remote interface is almost as easy to manipulate as the local interface, you must always consider network latency, network failure, and distributed systems failures. You must develop a significant amount of error detection and correction logic to anticipate these impacts. With high-latency network round trips, you want to avoid a large number of fine-grained method invocations. Therefore, use Remote Facade [Fowler03] and Data Transfer Object [Trowbridge03] to optimize call granularity and payload size.

This kind of integration works best when the systems that you want to connect are located in the same data center and when those systems have fairly reliable and high-speed connections. This kind of integration does not work well across slow and unreliable connections, including any connection that uses the Internet. This kind of integration is fragile if incompatible updates are introduced to any of the participants. Of course, if the target system you want to integrate with only exposes object-based APIs, you must use this method or an adapter to connect.

Choosing Message-Oriented Middleware Integration

Message-Oriented Middleware Integration is an excellent choice when systems are not reliably connected because there is the potential for network failure or distributed systems failure. Because Message-Oriented Middleware Integration works by placing messages asynchronously into a durable queue, there is little chance that the messages could be lost during a system or network failure. In addition, using asynchronous communication decouples the sender from the receiver. The application that sends the messages will continue to function even if the receiver fails, and the failed message receiver will continue processing messages from the queue after the receiver is restored.

The sender and the receivers are also decoupled from each other because they use a common message format. This makes it easier to separate the message from the set of intended receivers in the enterprise. After a message is sent from a source, you can design an integration network that uses the Message Broker pattern, the Message Bus pattern, the Publish-Subscribe pattern, or a host of other message receivers without modifying the original message sender. This allows the source, distribution, and consumption of the message to vary independently, thus improving flexibility.

Of course, there are tradeoffs involved with using Message-Oriented Middleware. Although the programming model is fairly simple for Distributed-Object Integration, the programming model becomes much more complex for Message-Oriented Middleware. Messages arriving at endpoints become events; therefore, the programming model is event driven. In addition, these events arrive across networks that may be unreliable and that do not have implicit correlation between them. The order of message arrival is not necessarily guaranteed and there may be duplicate messages. Sometimes, processing duplicate messages (also known as idempotent messages) does not adversely affect your system. Other times, processing duplicate messages is a serious flaw, for example, when you are transferring money. The security context of the message must be established, policies must be applied, and trust boundaries must be established. Your particular requirements must anticipate and justify this additional complexity. In addition, using proprietary message-oriented integration binds you to a particular message-oriented middleware implementation. It must be installed on the endpoint you want to communicate with, which may not always be the case inside your enterprise or between trading partners.

Service-Oriented Integration

Most enterprises have heterogeneous technical architectures and want to take advantage of system-based collaboration over the Internet. The most competitive enterprises want flexible and automated business processes that may require technology independence. All of these factors contribute to an urgent requirement for interoperability between different technical architectures. The best way to design for interoperability is by using Service-Oriented Integration.

Like Message-Oriented Middleware Integration, Service-Oriented Integration uses messages to communicate between senders and receivers. Unlike Message-Oriented Middleware, Service-Oriented Integration uses XML Schema and SOAP to transport and resolve messages. These two standards provide a portable type system and an extensible messaging framework that is not coupled to any proprietary implementation or transport. In addition, Service-Oriented Integration recommends the Web Services Integration (WS-I) Basic Profile to ensure interoperability between endpoints.

Service-Oriented Integration enables interoperability and allows you to send both synchronous and asynchronous messages (for more information, see the Service-Oriented Integration pattern). As a result, you can have the same kind of complex programming model as Message-Oriented Middleware. In addition, you have the complexity of a new and still emerging set of standards to understand and comply with. Therefore, make sure your requirements justify this level of complexity.

After you build these interoperable systems, there is one more tradeoff to consider: performance. Using XML incurs the cost of serializing, deserializing, and parsing XML documents. In addition, XML documents are often much larger than their binary equivalents because they contain metadata. This can increase the size of the payload that must be processed during message exchanges. However, because processing power is relatively inexpensive and because processors get faster every year, the issue of payload size can be addressed though hardware. Also, you can selectively trade interoperability for performance by using binary encoding inside your SOAP message as needed. Finally, given the support of major vendors, it is likely that new platform infrastructure will evolve in a way that will optimize Web services–based performance.

Combining Distributed Objects, Message-Oriented Middleware, and Services

It is likely you will use some combination of all three types of integration in your enterprise. Start by identifying services at a level of granularity that is meaningful to the business, such as steps within a business process. This will help you define service boundaries where interoperability is important. For interactions across these boundaries, use Service-Oriented Integration. For an example of service identification, see Chapter 9, "Integration Examples." To implement a service within these boundaries, you may want to use Distributed-Object Integration, Message-Oriented Middleware Integration, or Service-Oriented Integration. If you need to do two-phase commits across distributed databases, using Distributed-Object Integration permits you to take advantage of platform infrastructure that supports two-phase commits across distributed databases. Just ensure that you keep these transactions inside your service boundaries.

Any time you connect with systems and networks that you consider to be unreliable, consider using Message-Oriented Middleware to provide a durable buffer between connected systems. Using messages as your means of communication, either through Message-Oriented Middleware or Service-Oriented Integration, permits you to construct more advanced integration architectures that use Message Broker and Message Bus to communicate. As needed, you can always connect these integration architectures by using object-based APIs as described by the Adapter pattern [Gamma95].

Regardless of the kind of Functional Integration you use, you will encounter the following benefits and liabilities.

Benefits

• Flexibility. Functional Integration is very flexible. The abstraction, in the form of a function call, permits many different types of interchanges, such as data replication, shared business functions, or business process integration. This also implies that a generic mechanism for Functional Integration can be used in many different integration scenarios.
• Encapsulation. A functional interface provides an abstraction from an application's internal workings. This isolates users of the programming interface from variations in the application's internal workings or data representations.
• Robust. Executing a function through an application's business logic layer ensures that only well-defined functions can be executed. It also ensures that all validations built into the application logic are executed. This ensures that other applications cannot perform invalid functions or corrupt the application's internal state.
• Familiar programming model. Functional Integration provides a programming model that is more aligned with widespread application programming models. Functional Integration is therefore familiar to most application developers. Other styles of integration, such as Data Integration or Presentation Integration, require developers to learn a new style of development.

Liabilities

• Tighter coupling. One application that sends a command directly to another application results in tighter coupling compared to publishing data to a common Message Bus because the requesting application requires information about the location of the application that is providing the service. However, inserting a Message Broker can alleviate this type of coupling.
• Requires business layer to be exposed. Naturally, Functional Integration is limited to scenarios where the affected applications expose a suitable functional interface. This is the case for most contemporary business applications, but it may not be true for older monolithic applications or for Web applications hosted by external entities. In those cases, Presentation Integration may be the only option.
• Limited to available functions. In most cases, Functional Integration is limited to the functions that are already implemented in the application's business logic. Extending the application's logic with new functions usually involves significant development and may outweigh the other benefits of Functional Integration. In such cases, implementing the new function externally might be a more efficient approach.
• Inefficient with large datasets. Existing functional interfaces are generally designed to execute individual functions. This typically makes them inefficient for the transmission of large datasets because many individual requests must be made.
• Programming-language specific. Many functional interfaces are tied to a specific programming language or technology because the stronger semantics of the conversation require a more detailed contract that includes the representation of method names, complex data structures, return values, exceptions, and other elements.

Testing Considerations

Functional Integration is easier to test than other integration approaches for the following reasons:

• The additional abstraction provided by the separation of the exposed functional interface and the internal execution of the function permits the creation of mock objects and test stubs. These constructs work with the external applications in the same way the full application does because they implement the same interface. However, the test stub or mock object usually does not perform actual business functions that may be slow or that may require many external resources. Instead, the stub fabricates response data, or it logs data into a logging mechanism that can be monitored by the test tools.
• Because Functional Integration is based on the same programming model as the application itself, tests of the integration are easier to tie to the testing of the application itself. This type of testing is also generally well-supported by test tools such as NUnit that target application testing.

However, you must consider the following considerations. Your ability to test depends on how well the functional interface of the existing application is structured. If the interface exposes functions that are well-defined and that are free of side effects free, testing should be relatively easy. If the functional interface consists of a blur of poorly defined functions that are full of side effects, then testing will be difficult.

Security Considerations

The richer semantics of Functional Integration allow for a more finely grained security model. Many applications enforce security rules based on user identity, and they control permissions on an object level. For example, a user is only allowed to access the accounts that belong to that user and to access related accounts. This type of permission checking is typically not available with Data Integration.

The disadvantage of this more complex security model is that the integration solution has to acquire the proper user credentials to access data or to invoke functions. Because other applications are generally unaware of the security context used in another application, the security context has to be established by the middleware. You can establish the security context either by using a fixed user or by using an explicit translation of security contexts between applications.

Acknowledgments

[Hohpe04] Hohpe, Gregor, Bobby Woolf, Enterprise Integration Patterns. Addison-Wesley, 2004.

[Linthicum04] Linthicum, David. Next Generation Application Integration. Addison-Wesley, 2004.

[Ruh01] Ruh, William. Enterprise Application Integration. A Wiley Tech Brief. Wiley, 2001.

[Trowbridge03] Trowbridge, David, et al. Enterprise Solution Patterns Using Microsoft .NET. Microsoft Press, 2003. Also available on the MSDN Architecture Center at: http://msdn.microsoft.com/architecture/patterns/default.aspx?pull=/library/en-us/dnpatterns/html/Esp.asp.

[W3C04] "Web Services Architecture W3C Working Draft 11 February 2004." Available on the W3C Web site at: http://www.w3.org/TR/2004/NOTE-ws-arch-20040211/

Start | Previous | Next